Cooled electronic system with thermal spreaders coupling electronics cards to cold rails

ABSTRACT

Liquid-cooled electronic systems are provided which include an electronic assembly having an electronics card and a socket with a latch at one end. The latch facilitates securing of the card within the socket or removal of the card from the socket. A liquid-cooled cold rail is disposed at the one end of the socket, and a thermal spreader couples the electronics card to the cold rail. The thermal spreader includes first and second thermal transfer plates coupled to first and second surfaces on opposite sides of the card, and thermally conductive extensions extending from end edges of the plates, which couple the respective transfer plates to the liquid-cooled cold rail. The thermally conductive extensions are disposed to the sides of the latch, and the card is securable within or removable from the socket using the latch without removing the cold rail or the thermal spreader.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

This invention was made with Government support under Contract No.DE-EE0002894, awarded by the Department of Energy. Accordingly, the U.S.Government has certain rights in the invention.

BACKGROUND

As is known, operating electronic devices produce heat. This heat shouldbe removed from the devices in order to maintain device junctiontemperatures within desirable limits, with failure to remove heateffectively resulting in increased device temperatures, potentiallyleading to thermal runaway conditions. Several trends in the electronicsindustry have combined to increase the importance of thermal management,including heat removal for electronic devices, including technologieswhere thermal management has traditionally been less of a concern, suchas CMOS. In particular, the need for faster and more densely packedcircuits has had a direct impact on the importance of thermalmanagement. First, power dissipation, and therefore heat production,increases as device operating frequencies increase. Second, increasedoperating frequencies may be possible at lower device junctiontemperatures. Further, as more and more devices are packed onto a singlechip, heat flux (Watts/cm²) increases, resulting in the need to removemore power from a given size chip or module. These trends have combinedto create applications where it is no longer desirable to remove heatfrom modern devices solely by traditional air cooling methods, such asby using air cooled heat sinks with heat pipes or vapor chambers. Suchair cooling techniques are inherently limited in their ability toextract heat from an electronic device with high power density.

BRIEF SUMMARY

In one aspect, the shortcomings of the prior art are overcome andadditional advantages are provided through the provision of a cooledelectronic system which includes an electronic assembly, and a coolingapparatus. The electronic assembly includes an electronics card,comprising a first surface and a second surface on opposite sides of theelectronics card, and a socket with a latch at one end of the socket.The latch facilitates latching the electronics card within the socket orunlatching the electronics card from the socket. The cooling apparatusincludes a liquid-cooled structure and a thermal spreader. Theliquid-cooled structure is disposed at the one end of the socket, andcomprises a thermally conductive structure with at least onecoolant-carrying channel extending therein. The thermal spreaderfacilitates cooling the electronics card by coupling the card to theliquid-cooled structure. The thermal spreader includes first and secondthermal transfer plates, and first and second thermally conductiveextensions. The first thermal transfer plate and the second thermaltransfer plate respectively include first and second thermal conductionsurfaces which are spaced in opposing relation and configured toaccommodate the electronics card therebetween, with the first thermalconduction surface thermally coupled to the first surface of theelectronics card, and the second thermal conduction surface thermallycoupled to the second surface of the electronics card. The first thermaltransfer plate and the second thermal transfer plate each include afirst end edge and a second end edge. The first thermally conductiveextension extends from the first end edge of the first thermal transferplate, and the second thermally conductive extension extends from one ofthe first end edge or the second end edge of the second thermal transferplate. The first thermally conductive extension couples the firstthermal transfer plate to the liquid-cooled structure, and is disposedat a first side of the latch at the one end of the socket of theelectronic assembly, wherein the latch includes opposite first andsecond sides. The latch is accessible and facilitates securing of theelectronics card within the socket or removal of the electronics cardfrom the socket without removing at least one of the liquid-cooledstructure or the thermal spreader.

In another aspect, a liquid-cooled electronics rack is provided, whichincludes an electronics rack comprising an electronic assembly, and acooling apparatus. The electronic assembly includes an electronics card,comprising a first surface and a second surface on opposite sides of theelectronics card, and a socket with a latch at one end of the socket.The latch facilitates latching the electronics card within the socket orunlatching the electronics card from the socket. The cooling apparatusincludes a liquid-cooled structure and a thermal spreader. Theliquid-cooled structure is disposed at the one end of the socket, andcomprises a thermally conductive structure with at least onecoolant-carrying channel extending therein. The thermal spreaderfacilitates cooling the electronics card by coupling the card to theliquid-cooled structure. The thermal spreader includes first and secondtransfer plates, and first and second thermally conductive extensions.The first thermal transfer plate and the second thermal transfer platerespectively include first and second thermal conduction surfaces, whichare spaced in opposing relation and configured to accommodate theelectronics card therebetween, with the first thermal conduction surfacethermally coupled to the first surface of the electronics card, and thesecond thermal conduction surface thermally coupled to the secondsurface of the electronics card. The first thermal transfer plate andthe second thermal transfer plate each includes a first end edge and asecond end edge. The first thermally conductive extension extends fromthe first end edge of the first thermal transfer plate, and the secondthermally conductive extension extends from one of the first end edge orthe second end edge of the second thermal transfer plate. The firstthermally conductive extension couples the first thermal transfer plateto the liquid-cooled structure, and is disposed at a first side of thelatch at the one end of the socket of the electronic assembly, whereinthe latch includes opposite first and second sides. The latch isaccessible and facilitates securing of the electronics card within thesocket or removal of the electronics card from the socket withoutremoving at least one of the liquid-cooled structure or the thermalspreader.

In a further aspect, a method of fabricating a cooled electronic systemis provided. The method includes: providing an electronic assembly, theelectronic assembly including: an electronics card comprising a firstsurface and a second surface on opposite sides of the electronics card,and a socket with a latch at one end of the socket, the latchfacilitating latching of the electronics card within or unlatching ofthe electronics card from the socket; disposing a liquid-cooledstructure at the one end of the socket, the liquid-cooled structurecomprising a thermally conductive structure with at least onecoolant-carrying channel extending within the thermally conductivestructure; and providing a thermal spreader coupling the electronicscard to the liquid-cooled structure. The thermal spreader includes firstand second thermal transfer plates, and first and second thermallyconductive extensions. The first thermal transfer plate and the secondthermal transfer plate respectively include first and second thermalconduction surfaces, which are spaced in opposing relation andconfigured to accommodate the electronics card therebetween, with thefirst thermal conduction surface thermally coupled to the first surfaceof the electronics card, and the second thermal conduction surfacethermally coupled to the second surface of the electronics card. Thefirst thermal transfer plate and the second thermal transfer plate eachincludes a first end edge and a second end edge. The first thermallyconductive extension extends from the first end edge of the firstthermal transfer plate, and the second thermally conductive extensionextends from the first end edge of the second thermal transfer plate.The first thermally conductive extension couples the first thermaltransfer plate to the liquid-cooled structure, and the second thermallyconductive extension couples the second thermal transfer plate to theliquid-cooled structure, wherein the first thermally conductiveextension is disposed at a first side of the latch at the one end of thesocket of the electronic assembly, and the second thermally conductiveextension is disposed at a second side of the latch, the first side andthe second side of the latch being opposite first and second sides ofthe latch. The latch is accessible and facilitates securing of theelectronics card within the socket or removal of the electronics cardfrom the socket without removing of at least one of the liquid-cooledstructure or the thermal spreader.

Additional features and advantages are realized through the techniquesof the present invention. Other embodiments and aspects of the inventionare described in detail herein and are considered a part of the claimedinvention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

One or more aspects of the present invention are particularly pointedout and distinctly claimed as examples in the claims at the conclusionof the specification. The foregoing and other objects, features, andadvantages of the invention are apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 depicts one embodiment of a conventional raised floor layout ofan air-cooled data center;

FIG. 2 is a cross-sectional plan view of one embodiment of anelectronics rack with an attached air-to-liquid heat exchanger enhancingcooling of air passing through the electronics rack;

FIG. 3 depicts one embodiment of a data center with a coolantdistribution unit facilitating liquid-cooling of one or moreliquid-cooled electronics racks of the data center, in accordance withan aspect of the present invention;

FIG. 4 depicts an alternate embodiment of a cooling apparatus andliquid-cooled electronics rack, in accordance with one or more aspectsof the present invention;

FIG. 5A is a more detailed, elevational view of one embodiment of theliquid-cooled electronics rack of FIG. 4, and illustrating rack-levelcoolant distribution structures, in accordance with one or more aspectsof the present invention;

FIG. 5B is a partial depiction of a more detailed embodiment of therack-level coolant distribution structures illustrated in FIG. 5A, inaccordance with one or more aspects of the present invention;

FIG. 6 is a plan view of one embodiment of an electronic system layoutfor a liquid-cooled electronics rack, and illustrating multipleliquid-cooled cold plates and multiple liquid-cooled cold rails coupledin fluid communication, in accordance with one or more aspects of thepresent invention;

FIG. 7 depicts one detailed embodiment of a partially assembledelectronic system, wherein the electronic system includes multipledifferent types of heat-generating electronic devices to be cooled, inaccordance with one or more aspects of the present invention;

FIG. 8A depicts the electronic system of FIG. 7, with first, second andthird liquid-cooled cold rails of a cooling apparatus shown in place atthe ends of first and second arrays of sockets and electronics cards ofthe electronic system, in accordance with one or more aspects of thepresent invention;

FIG. 8B is a partial depiction of the partially assembled cooledelectronic system of FIG. 8A, and illustrating in greater detail, oneembodiment of the first liquid-cooled cold rail disposed at one end ofthe parallel-disposed sockets that form part of the first array ofelectronics cards, in accordance with one or more aspects of the presentinvention;

FIG. 8C is a partial depiction of the second liquid-cooled cold raildisposed at the other end of the sockets comprising part of the firstarray of electronics cards, and shown disposed between the first andsecond arrays of electronics cards illustrated in FIG. 8A, in accordancewith one or more aspects of the present invention;

FIG. 8D depicts one embodiment of the third liquid-cooled cold raildisposed at the other end of the sockets that form part of the secondarray of electronics cards illustrated in FIG. 8A, in accordance withone or more aspects of the present invention;

FIG. 9A depicts the partially assembled, cooled electronic system ofFIGS. 8A-8D, with a plurality of thermal spreaders shown coupled to theelectronics cards and thermally interfacing the electronics cards torespective liquid-cooled cold rails, in accordance with one or moreaspects of the present invention;

FIG. 9B is a partial depiction of the cooled electronic system of FIG.9A, and illustrating interfacing of thermal spreaders to the firstliquid-cooled cold rail at the one end of the sockets of the first arrayof electronics cards, in accordance with one or more aspects of thepresent invention;

FIG. 9C is a partial depiction of the cooled electronic system of FIG.9A, and illustrating the second liquid-cooled cold rail disposed betweenthe first and second arrays of electronics cards, and showinginterfacing of thermal spreaders coupled to the first array ofelectronics cards, and thermal spreaders coupled to the second array ofelectronics cards to the second liquid-cooled cold rail, in accordancewith one or more aspects of the present invention;

FIG. 9D is a partial depiction of the cooled electronic system of FIG.9A, and illustrating interfacing of thermal spreaders associated withthe second array of electronics cards to the third liquid-cooled coldrail of the cooled electronic system, in accordance with one or moreaspects of the present invention;

FIG. 10A depicts the cooled electronic system of FIGS. 7-9D, with athermal spreader shown clipped to an electronics card being insertedinto a respective socket of the first array of electronics cards, inaccordance with one or more aspects of the present invention;

FIG. 10B depicts the cooled electronic system of FIG. 10A, with theillustrated electronics card and associated thermal spreader shown inlatched position within the first array of electronics cards, inaccordance with one or more aspects of the present invention;

FIG. 11A depicts one embodiment of a thermal transfer plate of a heatspreader, similar to the heat spreader illustrated in FIGS. 9A-9D, inaccordance with one or more aspects of the present invention;

FIG. 11B depicts the thermal transfer plate of FIG. 11A, and illustratesan opposite side of the plate, the opposite side comprising a thermalconduction surface to be physically coupled to a first surface on oneside of a respective electronics card, in accordance with one or moreaspects of the present invention;

FIG. 11C depicts one embodiment of an assembled subassembly comprisingan electronics card, a thermal spreader, and spring clips shown forcingthe thermal transfer plates of the thermal spreader together and intophysical contact with surfaces on opposite sides of the electronicscard, in accordance with one or more aspects of the present invention;

FIG. 12A depicts an alternate embodiment of a thermal transfer plate ofa thermal spreader such as illustrated in FIGS. 9A-9D, and showncomprising multiple heat pipes extending within the thermal transferplates and thermally conductive extensions at the ends thereof, inaccordance with one or more aspects of the present invention;

FIG. 12B depicts an alternate embodiment of the thermal transfer plateof FIG. 12A, wherein a single heat pipe extends from one thermallyconductive extension at one end of the thermal transfer plate to theother thermally conductive extension at the other end of the thermaltransfer plate, in accordance with one or more aspects of the presentinvention;

FIG. 12C depicts another embodiment of a thermal spreader comprising afirst thermal transfer plate with a thermally conductive extension atone end thereof, configured for coupling to a first liquid-cooled coldrail, and a heat pipe extending through the first thermal transfer plateinto the thermally conductive extension, and a second thermal transferplate with a thermally conductive extension configured for coupling to asecond liquid-cooled cold rail, in accordance with one or more aspectsof the present invention;

FIG. 13 depicts another embodiment of a thermal spreader for a cooledelectronic system, such as illustrated in FIGS. 9A-9D, wherein multipleheat pipes are disposed at the outer-facing surfaces of the thermaltransfer plates of the heat spreader, and a single, elongate heat pipeis disposed at each of the inner-facing surfaces of the thermal transferplates of the thermal spreader, in accordance with one or more aspectsof the present invention;

FIG. 14A depicts another embodiment of a thermal spreader, such asillustrated in FIGS. 9A-9D, and showing a coupling mechanism couplingthe thermally conductive extensions of the thermal spreader to aliquid-cooled cold rail, in accordance with one or more aspects of thepresent invention;

FIGS. 14B-14D depict an alternate embodiment of a thermal spreader, suchas illustrated in FIGS. 9A-9D, and illustrating another couplingmechanism for coupling of the thermal spreader to a liquid-cooled coldrail, in accordance with one or more aspects of the present invention;

FIGS. 15A-15D illustrate one embodiment of a single-piece thermalspreader, and illustrating the single-piece thermal spreader and anelectronics card being assembled prior to insertion of the electronicscard into a respective socket of the electronic system, in accordancewith one or more aspects of the present invention;

FIG. 16A illustrates an alternate embodiment of a liquid-cooled coldrail for a cooled electronic system, in accordance with one or moreaspects of the present invention;

FIG. 16B depicts the liquid-cooled cold rail of FIG. 16A, andillustrates a plurality of thermal spreaders physically coupled to theliquid-cooled cold rail, in accordance with one or more aspects of thepresent invention;

FIG. 17A illustrates another coupling embodiment for interfacing thethermal spreaders to the liquid-cooled cold rail, wherein the connectingflange between the first and second thermal transfer plates of thethermal spreader is configured to reside within a slot structure formedin association with the upper surface of the liquid-cooled cold rail, inaccordance with one or more aspects of the present invention;

FIG. 17B depicts still another coupling embodiment for interfacing thethermal spreaders to the liquid-cooled cold rail, wherein the thermallyconductive extensions extending from the thermal transfer plates residebetween respective opposing fins projecting from the upper surface ofthe liquid-cooled cold rail, in accordance with one or more aspects ofthe present invention;

FIG. 18A depicts another embodiment of a thermal spreader for a cooledelectronic system, in accordance with one or more aspects of the presentinvention;

FIG. 18B depicts the thermal spreader of FIG. 18A, shown with anelectronics card disposed between the first and second thermal transferplates of the thermal spreader, in accordance with one or more aspectsof the present invention;

FIG. 18C is a partially assembled, cooled electronic system illustratingthe thermal spreaders of FIGS. 18A & 18B, coupling the respectiveelectronics cards to the illustrated liquid-cooled cold rail, inaccordance with one or more aspects of the present invention;

FIG. 19A depicts another embodiment of a thermal spreader, coupled to anelectronics card to be inserted within a cooled electronic system, inaccordance with one or more aspects of the present invention;

FIG. 19B is a partial depiction of a cooled electronic system, andillustrating a plurality of thermal spreaders such as depicted in FIG.19A, coupling respective electronics cards to a liquid-cooled cold rail,wherein the liquid-cooled cold rail has an angled upper surface, inaccordance with one or more aspects of the present invention; and

FIG. 19C is a top plan view of the partially assembled, cooledelectronic system of FIG. 19B, in accordance with one or more aspects ofthe present invention.

DETAILED DESCRIPTION

As used herein, the terms “electronics rack”, “rack-mounted electronicequipment”, and “rack unit” are used interchangeably, and unlessotherwise specified include any housing, frame, rack, compartment, bladeserver system, etc., having one or more heat generating components of acomputer system or electronics system, and may be, for example, astand-alone computer processor having high, mid or low end processingcapability. In one embodiment, an electronics rack may comprise aportion of an electronic system, a single electronic system or multipleelectronic systems, for example, in one or more sub-housings, blades,books, drawers, nodes, compartments, etc., having one or moreheat-generating electronic components disposed therein. An electronicsystem(s) within an electronics rack may be movable or fixed relative tothe electronics rack, with rack-mounted electronic drawers and blades ofa blade center system being two examples of electronic systems (orsubsystems) of an electronics rack to be cooled.

“Electronic component” refers to any heat-generating electroniccomponent of, for example, a computer system or other electronic systemrequiring cooling. By way of example, an electronic component maycomprise one or more integrated circuit dies, and/or other electronicdevices to be cooled, such as one or more electronics cards comprising aplurality of memory modules (such as one or more dual in-line memorymodules (DIMMs)).

Further, as used herein, the terms “liquid-cooled structure”,“liquid-cooled cold plate” and “liquid-cooled cold rail” refer tothermally conductive structures having one or more channels (orpassageways) formed therein or passing therethrough, which facilitatethe flow of liquid coolant through the structure. A liquid-cooledstructure may be, for example, a liquid-cooled cold plate or aliquid-cooled cold rail. In one example, tubing is provided extendingthrough the liquid-cooled structure. An “air-to-liquid heat exchanger”or “air-to-liquid heat exchange assembly” means any heat exchangemechanism characterized as described herein through which liquid coolantcan circulate; and includes, one or more discrete air-to-liquid heatexchangers coupled either in series or in parallel. An air-to-liquidheat exchanger may comprise, for example, one or more coolant flowpaths, formed of thermally conductive tubing (such as copper or othertubing) in thermal or mechanical contact with a plurality of air-cooledcooling fins. Size, configuration and construction of the air-to-liquidheat exchanger can vary without departing from the scope of theinvention disclosed. Still further, “data center” refers to a computerinstallation containing one or more electronics racks to be cooled. As aspecific example, a data center may comprise one or more rows ofrack-mounted computer units, such as server units.

One example of coolant used within the cooled electronic apparatusesdisclosed herein is water. However, the concepts presented are readilyadapted to use with other types of coolant. For example, the coolant maycomprise a brine, a fluorocarbon liquid, a liquid metal, or othersimilar coolant, or refrigerant, while still maintaining the advantagesand unique features of the present invention.

Reference is made below to the drawings, which are not drawn to scalefor reasons of understanding, wherein the same reference numbers usedthroughout different figures designate the same or similar components.

FIG. 1 depicts a raised floor layout of an air cooled data center 100typical in the prior art, wherein multiple electronics racks 110 aredisposed in one or more rows. A data center such as depicted in FIG. 1may house several hundred, or even several thousand microprocessors. Inthe arrangement illustrated, chilled air enters the computer room viaperforated floor tiles 160 from a supply air plenum 145 defined betweenthe raised floor 140 and a base or sub-floor 165 of the room. Cooled airis taken in through louvered covers at air inlet sides 120 of theelectronics racks and expelled through the back (i.e., air outlet sides130) of the electronics racks. Each electronics rack 110 may have one ormore air moving devices (e.g., fans or blowers) to provide forcedinlet-to-outlet airflow to cool the electronic devices within the rackunit. The supply air plenum 145 provides conditioned and cooled air tothe air-inlet sides of the electronics racks via perforated floor tiles160 disposed in a “cold” aisle of the computer installation. Theconditioned and cooled air is supplied to plenum 145 by one or more airconditioning units 150, also disposed within the data center 100. Roomair is taken into each air conditioning unit 150 near an upper portionthereof. This room air may comprise, in part, exhausted air from the“hot” aisles of the computer installation defined, for example, byopposing air outlet sides 130 of the electronics racks 110.

Due to ever-increasing air flow requirements through electronics racks,and the limits of air distribution within a typical data centerinstallation, liquid-based cooling is being combined with conventionalair-cooling. FIGS. 2-4 illustrate various embodiments of a data centerimplementation employing a liquid-based cooling system.

FIG. 2 depicts one rack-level liquid-cooling solution which utilizeschilled facility water to remove heat from the computer installationroom, thereby transferring the cooling burden from the air-conditioningunit(s) to the building's chilled water coolers. The embodiment depictedin FIG. 2 is described in detail in commonly assigned, U.S. Pat. No.6,775,137. Briefly summarized, facility-chilled water 200 circulatesthrough one or more liquid-to-liquid heat exchangers 210, coupled via asystem coolant loop 211, to individual electronics racks 220 within thecomputer room. Rack unit 220 includes one or more air-moving devices 230for moving air flow from an air inlet side to an air outlet side acrossone or more drawer units 240 containing heat-generating electroniccomponents to be cooled. In this embodiment, a front cover 250 attachedto the rack covers the air inlet side, a back cover 255 attached to therack covers the air outlet side, and a side car disposed adjacent to(and/or attached to) the rack includes a heat exchanger 260 for coolingair circulating through the rack unit. Further, in this embodiment, theliquid-to-liquid heat exchangers 210 are multiple computer roomwater-conditioning (CRWC) units which are coupled to receive buildingchilled facility water 200. The building chilled facility water is usedto cool the system coolant within system coolant loop 211, which iscirculating through air-to-liquid heat exchanger 260. The rack unit inthis example is assumed to comprise a substantially enclosed housing,wherein the same air circulates through the housing that passes acrossthe air-to-liquid heat exchanger 260. In this manner, heat generatedwithin the electronics rack is removed from the enclosed housing via thesystem coolant loop, and transferred to the facility coolant loop forremoval from the computer installation room.

FIG. 3 depicts another embodiment of a rack-level, liquid-coolingsolution, which again uses chilled facility water to remove heat fromthe computer installation room, thereby transferring the cooling burdenfrom the air-conditioning unit(s) to the building's chilled watercoolers. In this implementation, one embodiment of a coolantdistribution unit 300 for a data center is illustrated. Within coolantdistribution unit 300 is a power/control element 312, areservoir/expansion tank 313, a liquid-to-liquid heat exchanger 314, apump 315 (often accompanied by a redundant second pump), facility waterinlet 316 and outlet 317 supply pipes, a supply manifold 318 supplyingwater or system coolant to the electronics racks 110 via couplings 320and lines 322, and a return manifold 319 receiving water or systemcoolant from the electronics racks 110, via lines 323 and couplings 321.Each electronics rack includes (in one example) a power/control unit 330for the electronics rack, multiple electronic systems or subsystems 340,a system coolant supply manifold 350, and a system coolant returnmanifold 360. As shown, each electronics rack 110 is disposed on raisedfloor 140 of the data center with lines 322 providing system coolant tosystem coolant supply manifolds 350 and lines 323 facilitating return ofsystem coolant from system coolant return manifolds 360 being disposedin the supply air plenum beneath the raised floor.

In the embodiment illustrated, system coolant supply manifold 350provides system coolant to cooling apparatuses disposed within theelectronic systems or subsystems (for example, to liquid-cooled coldplates or cold rails) via flexible hose connections 351, which aredisposed between the supply manifold and the respective electronicsystems within the rack. Similarly, system coolant return manifold 360is coupled to the electronic systems via flexible hose connections 361.Quick connect couplings may be employed at the interface betweenflexible hoses 351, 361 and the individual electronic systems. By way ofexample, these quick connect couplings may comprise various types ofcommercially available couplings, such as those available from ColderProducts Company, of St. Paul, Minn., USA, or Parker Hannifin, ofCleveland, Ohio, USA.

Although not shown, electronics rack 110 may also include anair-to-liquid heat exchanger, for example, disposed at an air outletside thereof, which also receives system coolant from the system coolantsupply manifold 350 and returns system coolant to the system coolantreturn manifold 360.

FIG. 4 illustrates another embodiment of a liquid-cooled electronicsrack and cooling system therefor, in accordance with one or more aspectsof the present invention. In this embodiment, the electronics rack 400has a side car structure 410 associated therewith or attached thereto,which includes an air-to-liquid heat exchanger 415 through which aircirculates from an air outlet side of electronics rack 400 towards anair inlet side of electronics rack 400, for example, in a closed looppath in a manner similar to that illustrated above in connection withthe cooling implementation of FIG. 2. In this example, the coolingsystem comprises an economizer-based, warm-liquid coolant loop 420,which comprises multiple coolant tubes (or lines) connecting, in theexample depicted, air-to-liquid heat exchanger 415 in series fluidcommunication with a coolant supply manifold 430 associated withelectronics rack 400, and connecting in series fluid communication, acoolant return manifold 431 associated with electronics rack 400, acooling unit 440 of the cooling system, and air-to-liquid heat exchanger415.

As illustrated, coolant flowing through warm-liquid coolant loop 420,after circulating through air-to-liquid heat exchanger 415, flows viacoolant supply plenum 430 to one or more electronic systems ofelectronics rack 400, and in particular, one or more cold plates and/orcold rails 435 associated with the electronic systems, before returningvia coolant return manifold 431 to warm-liquid coolant loop 420, andsubsequently to a cooling unit 440 disposed (for example) outdoors fromthe data center. In the embodiment illustrated, cooling unit 440includes a filter 431 for filtering the circulating liquid coolant, acondenser (or air-to-liquid heat exchanger) 442 for removing heat fromthe liquid coolant, and a pump 443 for returning the liquid coolantthrough warm-liquid coolant loop 420 to air-to-liquid heat exchanger415, and subsequently to the liquid-cooled electronics rack 400. By wayof example, hose barb fittings 450 and quick disconnect couplings 455may be employed to facilitate assembly or disassembly of warm-liquidcoolant loop 420.

In one example of the warm coolant-cooling approach of FIG. 4, ambienttemperature might be 30° C., and coolant temperature 35° C. leaving theair-to-liquid heat exchanger 442 of the cooling unit. The cooledelectronic system depicted thus facilitates a chiller-less data center.Advantageously, such a liquid-cooling solution provides highly energyefficient cooling of the electronic systems of the electronics rack,using liquid (e.g., water), that is cooled via circulation through theair-to-liquid heat exchanger located outdoors (i.e., a dry cooler) withexternal ambient air being pumped through the dry cooler. Note that thiswarm coolant-cooling approach of FIG. 4 is presented by way of exampleonly. In alternate approaches, cold coolant-cooling could be substitutedfor the cooling unit 440 depicted in FIG. 4. Such cold coolant-coolingmight employ building chilled facility coolant to cool the liquidcoolant flowing through the liquid-cooled electronics rack, andassociated air-to-liquid heat exchanger (if present), in a manner suchas described above in connection with FIGS. 2 & 3.

FIGS. 5A & 5B depict in greater detail one embodiment of a liquid-cooledelectronics rack, such as depicted in FIG. 4, in accordance with one ormore aspects of the present invention. In this implementation,liquid-cooled electronics rack 400 comprises a plurality of electronicsystems 500, within which one or more electronic components are to beliquid-cooled via, for example, one or more cold plates or cold rails,as described below. The cooling system includes coolant loop 420 coupledin fluid communication with coolant supply manifold 430 and coolantreturn manifold 431, both of which may comprise vertically-orientedmanifolds attached to liquid-cooled electronics rack 400. In thisembodiment, the rack-level coolant distribution system further includesindividual node-level supply hoses 510 supplying coolant from coolantsupply manifold 430 to cold plates and cold rails within the electronicsystems 500. As shown in FIG. 5B, coolant supply manifold 430 may be (inone embodiment) a vertically-oriented manifold with a plurality ofcoupling connections 511 disposed along the manifold, one for eachelectronic system 500 having one or more electronic components to beliquid-cooled. Coolant leaves the individual electronic systems 500 vianode-level return hoses 520, which couple the individual electronicsystems (or nodes) to coolant return manifold 431, and hence, to coolantloop 420. In the embodiment illustrated in FIG. 4, relativelywarm-liquid coolant, such as water, is supplied from the cooling unit,either directly, or through one or more air-to-liquid heat exchanger(s)415 (of FIG. 4), and hot coolant is returned via the coolant returnmanifold to the cooling unit. In one embodiment of the rack-levelcoolant distribution system illustrated in FIGS. 5A & 5B, the node-levelsupply and return hoses 510, 520 are flexible hoses.

FIG. 6 illustrates one embodiment of a cooled electronic system 500component layout, wherein one or more air-moving devices 600 provideforced air flow 601 to cool multiple components 610 within electronicsystem 500. Cool air is taken in through a front 602 and exhausted out aback 603 of the electronic system (or drawer). The multiple componentsto be cooled include, for example, multiple processor modules to whichliquid-cooled cold plates 620 (of the liquid-based cooling apparatus)are coupled, as well as multiple arrays 631, 632 of electronics cards630 (e.g., memory modules such as dual in-line memory modules (DIMMs)),which are to be thermally coupled to one or more liquid-cooled coldrails 625. As used herein “thermally coupled” refers to a physicalthermal transport path being established between components, forexample, between an electronics card and a liquid-cooled cold rail forthe conduction of heat from one to the other.

The illustrated liquid-based cooling approach further includes multiplecoolant-carrying tubes connecting in fluid communication liquid-cooledcold plates 620 and liquid-cooled cold rails 625. These coolant-carryingtubes comprise (for example), a coolant supply tube 640, multiple bridgetubes 641, and a coolant return tube 642. In the embodiment illustrated,bridge tubes 641 connect one liquid-cooled cold rail 625 in seriesbetween the two liquid-cooled cold plates 620, and connect in paralleltwo additional liquid-cooled cold rails 625 between the secondliquid-cooled cold plate 620 and the coolant return tube 642. Note thatthis configuration is provided by way of example only. The conceptsdisclosed herein may be readily adapted to use with variousconfigurations of cooled electronic system layouts. Note also, that asdepicted herein, the liquid-cooled cold rails are elongate, thermallyconductive structures comprising one or more channels through whichliquid coolant passes, for example, via one or more tubes extendingthrough the structures. The liquid-cooled cold rails are disposed, inthe embodiment illustrated, at the ends of the two arrays (or banks)631, 632 of electronics cards 630, and multiple thermal spreaders areprovided coupling in thermal communication electronics cards 630 andliquid-cooled cold rails 625. Various such thermal spreaders arediscussed below with reference to FIGS. 8A-19C.

FIG. 7 depicts in greater detail one embodiment of an electronic systemlayout comprising a printed circuit board 700 with two processor modules710, each of which is to have a respective liquid-cooled cold plate of aliquid-based cooling system coupled thereto, and multiple arrays 721,722 of electronics cards 720, such as memory cards comprising memorymodules on opposite first and second sides thereof. Electronics cards720 are held in respective sockets 730, mounted to printed circuit board700, and latches 740 at the ends of sockets 730 facilitate securing (orremoving) of electronics cards 720 within (or from) the respectivesockets 730. The cooling apparatus embodiments described hereinbelowadvantageously facilitate liquid-cooling of electronics cards 720without interfering with an operator's access to latches 740 at the endsof sockets 730. In addition to existing component constraints on thesurface of printed circuit board 700, there is assumed to be negligiblespace between a cover (not shown) of the electronic system (e.g.,server), and the top edge surfaces 725 of electronics cards 720.

FIGS. 8A-8D depicts a partial assembly of a cooled electronic systemcomprising the electronic system layout of FIG. 7, including printedcircuit board 700, processor modules 710, and arrays 721, 722 ofelectronics cards 720. Electronics cards 720 are each shown positionedwithin a respective socket 730 mounted to printed circuit board 700,with latches 740 being disposed at the opposite ends of each socket 730.Latches 740 facilitate securing (or removing) electronics cards 720within (or from) the sockets.

FIGS. 8A-8D further depict multiple liquid-cooled cold rails 800, 810,820, shown positioned at the ends of the elongate sockets 730 of the twoarrays 721, 722 of electronics cards 720. Advantageously, theseliquid-cooled cold rails are configured and positioned to not interferewith opening and closing of latches 740. The multiple cold rails includea first liquid-cooled cold rail 800, disposed at one end of sockets 730in the first array 721 of electronics cards 720, a second liquid-cooledcold rail 810 disposed between the two arrays 721, 722 of electronicscards 720, and a third liquid-cooled cold rail 820 disposed at the otherend of sockets 730 of the second array 722 of electronics cards 720.Holes 830 are provided within each of the cold rails. In one embodiment,these holes may comprise threaded holes in the cold rails whichfacilitate attachment of the thermal spreaders (not shown) to therespective cold rails, as described further below.

In FIG. 8B, first liquid-cooled cold rail 800 is illustrated in greaterdetail at the one end of the sockets 730 of the first array 721 ofelectronics cards 720. As noted above, each cold rail is a thermallyconductive structure with at least one coolant-carrying channelextending therein. In this example, the coolant-carrying channel is aflattened tube 801 that is vertically-oriented within the cold rail andoffset from holes 830. As shown, first liquid-cooled cold rail 800 issized (in this example) to fit between sockets 730, and one or more fansockets 805. In addition, the cold rail may be selectively recessed atits bottom surface and/or one or more side surfaces to clear anyinterfering components, such as capacitors or chips, on the printedcircuit board. The flattened tube 801 extending through the thermallyconductive structure of the cold rail may comprise (as one example) aflattened ⅛ inch pipe, which may be routed above any intrusive elementson the board. In the example illustrated, quick disconnects 802 (FIG.8A) are shown provided at the ends of flattened tube 801.

In FIG. 8C, second liquid-cooled cold rail 810 is illustrated in greaterdetail. This cold rail is configured and sized to fit between the twoarrays 721, 722 of electronics cards. Two rows of holes 830 are providedwithin second liquid-cooled cold rail 810 to facilitate coupling ofthermal spreaders from the different arrays to the cold rail. In orderto clear the holes in the cold rail, a vertically-oriented, flattenedpipe 811 passing through cold rail 810 is positioned within a slot 813cut in the middle of the cold rail, for example, from the lower surfaceof the cold rail into the thermally conductive structure. By way ofexample, flattened tube 811 may be a flattened ⅛ inch pipe. Quickdisconnect couplings 802 may also be provided for connecting flattenedtube 811 in fluid communication with other coolant-carrying tubes, suchas the bridging tubes described above in connection with FIG. 6.

FIG. 8D illustrates in greater detail one embodiment of thirdliquid-cooled cold rail 820 disposed at the other end of the secondarray 722 of electronics cards 720. As illustrated, third liquid-cooledcold rail 820 is positioned to not interfere with opening or closing oflatches 740 at the other end of sockets 720 in the second array 722. Thecold rail includes a series of holes 830, which will facilitate couplingthermal spreaders (not shown) to the cold rail, and accommodates aflattened tube 821, which is vertically aligned within anappropriately-sized slot 823 in the thermally conductive structure ofthe cold rail and is offset from the series of holes 830. This cold railmay again be selectively recessed at its lower surface and/or sidesurfaces to clear any interfering components on printed circuit board700.

In the example of FIG. 8D, and assuming the cooling implementationdepicted in FIG. 6, the tube through which water flows may be aflattened, ¼ inch pipe, routed away from any intrusive elements on theprinted circuit board. As illustrated in FIG. 8A, quick disconnectcouplings 802 may be provided at the ends of flattened tube 820 tofacilitate coupling of the tube in fluid communication with other tubesof the liquid-based cooling approach discussed above in connection withFIG. 6. As illustrated in FIGS. 8A-8D, each liquid-cooled cold rail 800,810, 820 may be unique in terms of its location on the circuit board,and uniquely configured due to existing constraints within the differentareas of the printed circuit board. These liquid-cooled cold rails are,in one embodiment, coupled to either a cold liquid cooling loop or awarm-liquid cooling loop, depending on the cooling approach desired, asdescribed above.

FIGS. 9A-9D depict one embodiment of a cooled electronic systemcomprising the electronic subassembly of FIGS. 8A-8D, with a pluralityof thermal spreaders shown positioned between and in physical andthermal contact with the electronics cards of the arrays (or banks) ofelectronics cards. These thermal spreaders provide a thermal coupling orthermal conduction path from the electronics cards, for example, thememory modules on the opposite sides of the electronics cards, to theliquid-cooled cold rails to facilitate cooling of the electronics cardsvia conductive heat transfer to the cold rails, and hence to the liquidflowing through the cold rails.

In the embodiment illustrated, each thermal spreader comprises a firstthermal transfer plate 910 and a second thermal transfer plate 920. Thefirst thermal transfer plate comprises a first thermal conductionsurface, and the second thermal transfer plate 920 comprises a secondthermal conduction surface. The first thermal conduction surface and thesecond thermal conduction surface are in spaced, opposing relation, andare configured to accommodate a respective electronics card 720therebetween, with the first thermal conduction surface physically andthermally coupled to at least one first surface on one side of theelectronics card 720, and the second thermal conduction surfacephysically and thermally coupled to at least one second surface on theother side of the electronics card 720. These first and second surfaceson the different sides of the electronics card may comprise, in oneexample, surfaces of one or more electronics devices, such as memorymodules, mounted on the different sides of the respective electronicscard.

Further, the first thermal transfer plate 910 and second thermaltransfer plate 920 each comprise a first end edge 915, and a second endedge 916, disposed at opposite ends of the respective socket 730. Eachthermal transfer plate is a thermally conductive structure formed (inone example) as an elongate, flat plate. In this example, thermallyconductive extensions 912, 922 and 913, 923 are provided extending fromthe first and second end edges 915, 916 of each thermal transfer plate910, 920.

In one embodiment, these extensions 912, 922 and 913, 923 are curvedextensions, which may be characterized, in one embodiment, as “elephanttrunk-shaped extensions”. In particular, a first thermally conductiveextension 912 is a curved extension which extends from and upper portionof first thermal transfer plate 910 at the first end edge thereof 915,and a similar, second thermally conductive extension 922 extends fromthe first end edge 915 of second thermal transfer plate 920. Inaddition, a third thermally conductive extension 913 extends from thesecond end edge 916 of first thermal transfer plate 910, and a fourththermally conductive extension 923 extends from the second end edge 916of second thermal transfer plate 920. The thermally conductiveextensions 912, 922 at the first end edge 915 of the first and secondthermal transfer plates 910, 920 are spaced apart to allow access to therespective socket latch at the end of the socket 730 containing theelectronics card 720 sandwiched by the plates of the thermal spreader.Similarly, the thermally conductive extensions 913, 923 at the secondend edges 916 of the first and second thermal transfer plates 910, 920are spaced apart to allow access to the latch disposed at the other endof the socket. In this embodiment, the extensions 912, 922 and 913, 923are joined at their ends, and connected to the respective cold rail byrespective connecting flanges 930, each of which includes an opening935, aligned to an underlying opening 830 in the adjacent cold rail 800,810, 820. FIGS. 9B-9D illustrate these structures in greater detail,with the thermal spreaders 900 shown ready to be fastened to therespective cold rails using, for example, a threaded fastener.

As explained above, heat is transferred from the heat-generatingcomponents of the electronics card (for example, memory modules) to theconduction surfaces of the thermal transfer plates, across the thermaltransfer plates to the thermally conductive extensions at the endsthereof, and from the thermally conductive extensions into therespective liquid-cooled cold rails. From the liquid-cooled cold rails,the heat is rejected to coolant flowing through the channels or tubesextending through the cold rails, and subsequently, is removed from thecooled electronic system in a manner such as, for example, explainedabove in connection with FIGS. 4-6.

Note that in the embodiment depicted in FIGS. 9A-9D, the connectingflanges 930 at the ends of the thermally conductive extensions (wherecontacting the respective cold rails), are solid connecting structures,meaning that the thermal spreaders are (in one embodiment) single-piecestructures. Also, note that, in the approach depicted, heat is conductedby the thermal transfer plates to each end edge of the plates, and inparticular, to the thermally conductive extensions extending from therespective end edges to the two cold rails disposed at the opposite endsof the respective sockets in an array (or bank) of electronics cards.These thermally conductive extensions and connecting flanges physicallyand thermally couple to the upper surface of the respective cold rails.As illustrated in the plan views of FIGS. 9B-9D, the latches for therespective sockets remain accessible by appropriately spacing each pairof thermally conductive extensions to the first and second sides of thelatches at issue. This can be accomplished, in part, by reducing thethickness of the extensions compared with the thickness of the plates,as shown in the plan views of FIGS. 9B-9D.

FIGS. 10A-11C depict an alternate embodiment of a thermal spreader,which may be employed in a cooled electronic system such as depicted inFIGS. 9A-9D.

In FIGS. 10A & 10B, the cooled electronic system of FIGS. 9A-9D is againillustrated, wherein the first, second and third liquid-cooled coldrails 800, 810, 820 are shown in position, and a plurality of thermalspreaders 1000 are illustrated positioned about respective electronicscards of the electronic system. In this embodiment, the thermalspreaders are divided in half, thus comprising two separate thermaltransfer plates, each configured substantially as described above inconnection with the embodiment of FIGS. 9A-9D. In this embodiment,however, the thermal spreaders lack the connecting flanges at the endsof the thermally conductive extensions. Instead, two separate flanges(discussed below) extend from the respective thermally conductiveextensions inwards towards the other extension. These flanges are sizedand configured (in one embodiment) to mate when the thermal spreader isassembled about the respective electronics card. Aligning of the thermaltransfer plates, as well as aligning of the thermal transfer plates andthe electronics card, can be facilitated by providing appropriatealignment features on the plates and card. The flanges may also becurved at their ends to define an opening when positioned, asillustrated in FIGS. 10A-10B. The two halves (or two thermal transferplates) of the thermal spreader 1000 are clipped together, with eachshown positioned in physical and thermal contact with one side of therespective electronics card. In this embodiment, multiple retainingclips 1030 are illustrated. These retaining clips 1030 hold the platesof the thermal spreader in physical contact with respective sides of theelectronics card during assembly of the cooled electronic system. Asexplained above, the thermally conductive extensions from the ends ofthe thermal spreader are spaced and configured to not to interfere withnormal operation of the latches at the ends of the sockets within whichthe electronics cards are positioned. Each latch is thus accessible andfacilitates securing of the electronics card within the socket orremoval of the electronics card from the socket without requiringremoval of the liquid-cooled cold rail, or without requiring removal ofthe thermal spreader.

In FIGS. 11A & 11B, one thermal transfer plate (i.e., half of thethermal spreader 1000 of FIGS. 10A & 10B) is illustrated. Referringcollectively to FIGS. 11A & 11B, thermal transfer plate 1010 includes afirst thermal conduction surface 1100, configured to physically andthermally couple to one or more surfaces of one or more heat-generatingcomponents on one side of the electronics card, for example, one or morememory modules disposed on one side of the electronics card. Extendingfrom a first end edge 1101 of thermal transfer plate 1010 is a firstthermally conductive extension 1110, and extending from the other endedge 1102 of thermal transfer plate 1010 is a second thermallyconductive extension 1120. As noted above, in the one embodiment, theseextensions are curved, for example, to avoid contact with the respectivesocket within which the electronics card is to be positioned. Flanges1103, 1104 respectively extend inwards from thermally conductiveextensions 1110, 1120. In the illustrated embodiment, these flanges1103, 1104 are curved at their ends to define half of an opening sizedand configured to align to an opening in the respective liquid-cooledcold rail to which the thermally conductive extensions and flanges ofthe thermal spreader are to physically couple. The other half of thethermal spreader is a mirror image of thermal transfer plate 1010, suchthat the two thermal transfer plates have opposing first and secondthermal conduction surfaces, between which the respective electronicscard is sandwiched (as illustrated in FIG. 11C). The thermal transferplates and flanges are configured with appropriate thicknesses anddimensions so that when the thermal transfer plates sandwich theelectronics card, the flanges from the thermally conductive extensionsat respective ends of the card meet and their curved surfaces define theopenings that align with the openings in the adjacent liquid-cooled coldrail. As illustrated in this configuration, the thickness of the thermaltransfer plates may be different from that of the thermally conductiveextensions, for example, to facilitate providing sufficient room aroundthe corresponding latches to allow user access and operation of thelatches, without requiring disassembly of the thermal spreader.

FIG. 11C illustrates the assembled thermal spreader held in place aboutthe electronics card via an appropriate clipping mechanism, such asspring clips 1130. As illustrated in this figure, a lower portion of theelectronics card remains exposed to facilitate insertion of theelectronics card within the respective socket on the printed circuitboard of the electronic system. Also note that (in one embodiment), thethermal spreaders may be fabricated of a metal material, such as copper.

FIGS. 12A-12C illustrate various alternate embodiments of the thermaltransfer plates described above. In these alternate embodiments, one ormore flat heat pipes are employed within the thermal transfer plates andthe thermally conductive extensions from the transfer plates.

In FIG. 12A, a thermal transfer plate 1200 is depicted which is similarto thermal transfer plate 1010, described above in connection with FIGS.11A-11C. However, in this embodiment, two heat pipes 1210, 1220 areprovided extending from within the thermal transfer plate intorespective thermally conductive extensions at the end edges thereof. Inone embodiment, these heat pipes are flattened heat pipes, which arerecessed into, for example, the outer facing surface of the thermaltransfer plates and thermally conductive extensions. The heat pipes maybe provided where needed to enhance thermal performance of the thermalspreader.

In the alternate embodiment of FIG. 12B, a single heat pipe 1230 isillustrated extending through thermal transfer plate 1201 and intothermally conductive extensions from the end edges thereof. This singleheat pipe 1230 could be provided within an appropriate recess formedwithin the thermal transfer plate and thermally conductive extensions,for example, at the outer facing surfaces of the thermal transfer plateand extensions. Note that, in this discussion, “outer facing” refers tothat surface of the thermal transfer plate facing away from therespective electronics card to be cooled, and “inner facing” refers tothat surface of the thermal transfer plate facing the electronics cardto be cooled.

In FIG. 12C, one embodiment of an asymmetrical thermal spreader 1250 isillustrated. In this embodiment, heat from the different sides of asingle electronics card is transferred to different liquid-cooled coldrail. In particular, thermal spreader 1250 includes a first thermaltransfer plate 1260 and a second thermal transfer plate 1270, whichcomprise a first thermal conduction surface 1261 and a second thermalconduction surface 1271 in spaced, opposing relation. These thermalconduction surfaces of the thermal transfer plates are sized andconfigured to accommodate a single electronics card therebetween, withthe first thermal conduction surface thermally coupled to at least onefirst surface on one side of the electronics card, and the secondthermal conduction surface thermally coupled to at least one secondsurface on the other side of the electronics card. As shown, a firstthermally conductive extension 1265 extends from one end edge 1262 offirst thermal transfer plate 1260, and a second thermally conductiveextension 1275 extends from a different end edge 1272 of the secondthermal transfer plate 1270. Both thermally conductive extensions 1265,1275 are provided with flanges 1266, 1276, which extend inwards, in adirection towards the other plate. Thus (in this embodiment), the firstthermal transfer plate and first thermally conductive extension conductheat to a first liquid-cooled cold rail (not shown), and the secondthermal transfer plate and second thermally conductive extension conductheat to a different, second liquid-cooled cold rail (not shown).Securing of the thermal spreader 1250 to the two liquid-cooled coldrails is facilitated via providing openings 1267, 1277 in the respectiveflanges 1266, 1276 at the ends of the thermally conductive extensions.

As also shown, thermal spreader 1250 illustrated in FIG. 12C includes aheat pipe 1263 provided within, for example, at least one of the thermaltransfer plates extending to the respective thermally conductiveextension at the one end thereof. In one embodiment, both thermaltransfer plates and their extensions could comprise a flattened heatpipe on the outer surface thereof, as illustrated in FIG. 12C.

FIG. 13 depicts another embodiment of a thermal spreader 1300, inaccordance with an aspect of the present invention. This thermalspreader 1300 is substantially identical to the thermal spreaderdescribed above in connection with FIG. 12A, with the exception that theinner facing surfaces 1311, 1321 of the thermal transfer plates 1310,1320 may also comprise a heat pipe 1325, such as a flattened heat pipeprovided within a recess within the first and second thermal conductionsurfaces 1311, 1321 of the thermal transfer plates 1310, 1320.

FIG. 14A illustrates an alternate approach for assembling a two-partthermal spreader about an electronics card (not shown). In thisembodiment, the thermal spreader 1400 includes two thermal transferplates (such as described above) with first and second thermallyconductive extensions 1410, 1420 from the end edges thereof at one sideof the respective socket (not shown) mounted to the printed circuitboard. In this embodiment, a spring 1401, such as a compression spring,may be provided to connect the two thermal transfer plates. The springmay be squeezed around the electronics card during installation, and thethermal transfer plates may be bolted 1430, for example, using aclamping bolt, to the cold rail 1405. Thermal interfacing of thethermally conductive extensions 1410, 1420 to the cold rail 1405 isfacilitated by providing a thermal interface material 1402 between theextensions and the cold rail. As illustrated, the liquid-cooled coldrail includes one or more coolant-carrying channels 1406 extendingthrough the cold rail. Inward-extending flanges 1411, 1421 are providedat the ends of the thermally conductive extensions 1410, 1420, where theextensions meet the cold rail. These flanges define anappropriately-sized opening to accommodate the clamping bolt (or washer)1430, which holds the two halves of the thermal spreader securely inplace about the electronics card, particularly when repeated at oppositeends of the socket to clamp the thermal spreader to two different coldrails.

FIGS. 14B-14D depict an alternate embodiment for assembling a two-piecethermal spreader 1450 about an electronics card (not shown). In thisembodiment, the thermal spreader 1450 includes a first thermal transferplate and a second thermal transfer plate, each of which comprises athermally conductive extension 1460, 1470 extending from one end edgethereof at one end of the respective socket (not shown). A spring 1451,such as a compression spring, connects the two halves of the thermalspreader together, and a clip feature 1480 is also employed inconnection with the thermal transfer plates or the thermally conductiveextensions to facilitate holding the two halves of the thermal spreadertogether around the electronics card. The thermally conductiveextensions are shown bolted 1490 to the upper surface of liquid-cooledcold rail 1405, which as noted, comprises one or more liquidcoolant-carrying channels 1406. Aligned openings are provided inliquid-cooled cold rail 1405 and flanges 1461, 1471 at the ends of thethermally conductive extensions 1460, 1470, respectively, contacting theupper surface of the cold rail. A thermal interface material 1402 isprovided between the ends of the thermally conductive extensions and theupper surface of the cold rail to facilitate transfer of heat from theextensions into the cold rail.

In FIG. 14C, the clip feature 1480 is shown employed to secure the twohalves of the thermal spreader together, and in FIG. 14D, the clampingbolt (or washer) 1490 his shown threaded into position within thealigned openings to secure the thermal spreader in place about theelectronics card (not shown) and to the liquid-cooled cold rail.

FIGS. 15A-15D further depict a one-piece thermal spreader embodiment,such as described above in connection with FIGS. 9A-9D. In thissingle-piece thermal spreader embodiment, connecting flanges 930interconnect the ends of respective thermally conductive extensions 912,922, and 913, 923, that is, where the ends couple to the liquid-cooledcold rails. With this thermal spreader configuration, the spreader andelectronics card can be assembled either before the electronics card isinstalled into the respective socket on the printed circuit board, orafter. In one embodiment, assembly of the thermal spreader andelectronics card is facilitated by the use of opening clips 1500, withthe connecting flanges 930 between the extensions acting as springs incompression. As illustrated in FIG. 15B, the opening clips 1500 may beemployed during assembly to widen the space between the thermal transferplates 910, 920, after which the thermal spreader is positioned aboutthe electronics card (e.g., DIMM), as shown in FIG. 15C. Finally,opening clips 1500 may be removed (after releasing the compressionspring force), and the thermal spreader's spring action results insecure, physical coupling of the thermal transfer plates of the spreaderto the electronics card, thereby establishing the desired physical andthermal coupling between the inward-facing thermal conduction surfacesof the thermal transfer plates and the surfaces of the electronics card(e.g., modules on the opposite sides of the electronics card).

FIGS. 16A & 16B depict an alternate implementation for attaching thermalspreaders, such as described above in connection with FIGS. 9A-9D, tothe liquid-cooled cold rails. In this embodiment, the upper surface of aliquid-cooled cold rail 1600 is provided with a plurality of posts 1610projecting from the upper surface. These posts may comprise, in oneembodiment, threaded studs, over which the openings 935 in theconnecting flanges 930 of the thermal spreaders 900 are lowered. Oncepositioned as illustrated in FIG. 16B, a fastening mechanism, such as anut, could be used to secure the thermal spreader to the liquid-cooledcold rail.

In FIG. 17A, a further embodiment is presented for attaching the thermalspreaders, and more particularly, the thermally conductive extensions ofthe thermal transfer plates to the liquid-cooled cold rail. In thisembodiment, a slot structure is defined at the upper surface of the coldrail 1700 via, for example, an angled structure 1710 extending from theupper surface of the cold rail 1700 for the length of the cold rail. Inaddition, the connecting flanges 1720 at the ends of respectivethermally conductive extensions at a common end edge of the thermaltransfer plates, extend outwards from the thermally conductiveextensions in a manner sized and configured for insertion of theconnecting flanges within slot 1701, defined by the angled structure1710 and the upper surface of the liquid-cooled cold rail 1700, asillustrated.

FIG. 17B depicts another alternate embodiment for attaching thethermally conductive extensions of a thermal spreader to a liquid-cooledcold rail. In this embodiment, rather than providing flanges at the endsof the thermally conductive extensions for coupling to the upper surfaceof the liquid-cooled cold rail, a plurality of opposing fins 1730 areprovided projecting from the upper surface of the liquid-cooled coldrail. These opposing fins 1730 are spaced and configured to securelyreceive a respective thermally conductive extension of the thermalspreader by, for example, press-fitting the thermally conductiveextension into the space defined between adjacent opposing plate fins,as illustrated in FIG. 17B. Also, note that in the embodiments of FIGS.17A & 17B, the flattened tube extending through the liquid-cooled coldrail is disposed (by way of example) horizontally within theliquid-cooled cold rail in an upper portion of the liquid-cooled coldrail near the upper surface of the cold rail. This positioning mayenhance thermal performance. Additionally, note that in the embodimentsof FIGS. 17A & 17B, greater volume is available within the cold rail forlocating the coolant-carrying tube(s), and thus, larger tubes may beused with higher thermal performance and lower pressure dropcharacteristics.

FIGS. 18A-18C depict another implementation of a thermal spreader, inaccordance with an aspect of the present invention. In thisimplementation, the thermal spreader 1800 is one-sided, in that all ofthe heat conducted away from the electronics card 1801 by the spreaderis removed to one liquid-cooled cold rail disposed at one end of thearray of sockets containing the electronics cards. Aside from havingthermally conductive extensions 1810, 1820 from only one end edge of thethermal transfer plates 1811, 1821, respectively, this configuration issimilar to that described above in connection with FIGS. 11A-11C. Inparticular, thermal transfer plate 1811 comprises first thermallyconductive extension 1810 extending from a first end edge 1815 of thethermal transfer plate, and second thermal transfer plate 1821 includesa thermally conductive extension 1820 extending from one end edge 1825of the thermal transfer plate. These thermally conductive extensions arein spaced, opposing relation, and sized and configured to allow accessto the latches 1803 at the one end of sockets 1802. As illustrated,thermal spreader 1800 comprises two halves with flanges 1812, 1822extending from the ends of the thermally conductive extensions 1810,1820, that are sized to mate and configured to physically and thermallycouple to the upper surface 1804 of liquid-cooled cold rail 1805. Theflanges 1812, 1822 are curved where meeting to define an opening 1830,which aligns with a corresponding opening in the upper surface 1804 ofliquid-cooled cold rail 1805. A threaded attachment mechanism, such as ascrew or bolt, is subsequently employed to securely fasten the thermalspreader to the liquid-cooled cold plate.

FIGS. 19A-19C depict a further alternate embodiment of a thermalspreader and liquid-cooled cold plate, in accordance with an aspect ofthe present invention. In this embodiment, thermal spreader 1900 issimilar to the thermal spreader described above in connection with FIGS.11A-11C, except that the thermally conductive extensions from the endedges of the first and second thermal transfer plates are truncated andprovided with a 45° angle at their ends 1901, 1902. This 45° anglefacilitates mating the thermally conductive extensions to an angledupper surface of a liquid-cooled cold rail 1910, such as illustrated inFIG. 19B. In this configuration, the liquid-cooled cold rail has anupper surface with an inverted V-shape, which allows for coupling ofthermal spreaders from two different arrays of electronics cards of theelectronic system. The angled upper surface 1911 of liquid-cooled coldrail 1910 increases the contact area at the interface between thethermally conductive extensions from the thermal transfer plates and theupper surface of the cold rail. As in the above-described embodiments,flanges are provided at the ends of the thermally conductive extensions,which in one embodiment, define an opening aligned with a correspondingopening in the liquid-cooled cold rail, that may be employed tomechanically couple the thermal spreader to the cold rail, that is, viaan appropriate attachment mechanism. As illustrated in FIG. 19C, thelatches remain accessible and allow securing of the electronics card tothe socket or removal of the electronics card from the socket withoutrequiring removal of the liquid-cooled cold rail, or removal of thethermal spreader associated with the electronics card.

Although embodiments have been depicted and described in detail herein,it will be apparent to those skilled in the relevant art that variousmodifications, additions, substitutions and the like can be made withoutdeparting from the spirit of the invention and these are thereforeconsidered to be within the scope of the invention as defined in thefollowing claims.

1. A cooled electronic system comprising: an electronic assembly, theelectronic assembly comprising: an electronics card comprising a firstsurface and a second surface on opposite sides of the electronics card;a socket with a latch at one end of the socket, the latch facilitatinglatching of the electronics card within the socket or unlatching of theelectronics card from the socket; a cooling apparatus, the coolingapparatus comprising: a liquid-cooled structure disposed at the one endof the socket, the liquid-cooled structure comprising a thermallyconductive structure with at least one coolant-carrying channelextending therein; and a thermal spreader coupling the electronics cardto the liquid-cooled structure and facilitating conduction of heat fromthe electronics card to the liquid-cooled structure, the thermalspreader comprising: a first thermal transfer plate and a second thermaltransfer plate, the first thermal transfer plate comprising a firstthermal conduction surface and the second thermal transfer platecomprising a second thermal conduction surface, the first thermalconduction surface and the second thermal conduction surface being inspaced, opposing relation and configured to accommodate the electronicscard therebetween with the first thermal conduction surface coupled tothe first surface of the electronics card and the second thermalconduction surface coupled to the second surface of the electronicscard, and wherein the first thermal transfer plate and the secondthermal transfer plate each comprises a first end edge and a second endedge; and a first thermally conductive extension extending from thefirst end edge of the first thermal transfer plate, and a secondthermally conductive extension extending from one of the first end edgeor the second end edge of the second thermal transfer plate, the firstthermally conductive extension coupling the first thermal transfer plateto the liquid-cooled structure, and wherein the first thermallyconductive extension is disposed at a first side of the latch at the oneend of the socket of the electronic assembly, the latch comprisingopposite first, and second sides, and wherein the latch is accessibleand facilitates securing the electronics card within the socket orremoval of the electronics card from the socket without removing atleast one of the liquid-cooled structure or the thermal spreader.
 2. Thecooled electronic system of claim 1, further comprising anotherliquid-cooled structure disposed at another end of the socket, andwherein the second thermally conductive extension extends from thesecond end edge of the second thermal transfer plate and couples thesecond thermal transfer plate to the another liquid-cooled structure. 3.The cooled electronic system of claim 1, wherein the second thermallyconductive extension extends from the first end edge of the secondthermal transfer plate and couples the second thermal transfer plate tothe liquid-cooled structure, and wherein the second thermally conductiveextension is disposed at the second side of the latch at the one end ofthe socket of the electronic assembly.
 4. The cooled electronic systemof claim 3, wherein the first thermally conductive extension extendingfrom the first thermal transfer plate and the second thermallyconductive extension extending from the second thermal transfer plateeach physically couple to an upper surface of the liquid-cooledstructure.
 5. The cooled electronic system of claim 4, wherein the firstthermally conductive extension extending from the first thermal transferplate and the second thermally conductive extension extending from thesecond thermal transfer plate each comprise a flange extending from anend thereof physically coupled to the upper surface of the liquid-cooledstructure, the flange extending from the first thermally conductiveextension extending towards the second thermally conductive extension,and the flange extending from the second thermally conductive extensionextending towards the first thermally conductive extension, and whereinthe liquid-cooled structure comprises a post projecting from the uppersurface of the liquid-cooled structure, the flanges extending from thefirst and second thermally conductive extensions at least partiallyencircling the post projecting from the upper surface of theliquid-cooled structure.
 6. The cooled electronic system of claim 4,wherein the thermal spreader further comprises a compression springcoupling the first thermal transfer plate and the second thermaltransfer plate together, and a clamping bolt securing the firstthermally conductive extension and the second thermally conductiveextension together, and securing the first thermally conductiveextension and the second thermally conductive extension to theliquid-cooled structure.
 7. The cooled electronic system of claim 5,wherein the thermal spreader further comprises a clip mechanism securingthe first thermal transfer plate and the second thermal transfer platetogether, and securing the first thermal transfer plate and the secondthermal transfer plate in physical contact with the first surface andthe second surface, respectively, of the electronics card.
 8. The cooledelectronic system of claim 4, wherein the thermal spreader furthercomprises a connecting flange connecting the first thermally conductiveextension and the second thermally conductive extension where physicallyattached to the upper surface of the liquid-cooled structure, andwherein at least one of the connecting flange or the upper surface ofthe liquid-cooled structure includes an opening facilitating securingthe connecting flange to the upper surface of the liquid-cooledstructure.
 9. The cooled electronic system of claim 8, wherein theopening is disposed in the upper surface of the liquid-cooled structure,and wherein the at least one coolant-carrying channel in theliquid-cooled structure comprises a flat tube extending through theliquid-cooled structure, the flat tube being vertically oriented withinthe liquid-cooled structure and offset from the opening in the uppersurface of the liquid-cooled structure.
 10. The cooled electronic systemof claim 8, wherein the connecting flange and the upper surface of theliquid-cooled cold rail comprise aligned openings which facilitatesecuring the connecting flange to the upper surface of the liquid-cooledstructure.
 11. The cooled electronic system of claim 4, wherein thefirst thermally conductive extension is a first curved extensionextending from an upper portion of the first thermal transfer plate, andthe second thermally conductive extension is a second curved extensionextending from an upper portion of the second thermal transfer plate.12. The cooled electronic system of claim 11, wherein the thermalspreader further comprises a first heat pipe and a second heat pipe, thefirst heat pipe being disposed within the first thermal transfer plateand extending into the first curved extension, and the second heat pipebeing disposed within the second thermal transfer plate and extendinginto the second curved extension, the first heat pipe and the secondheat pipe facilitating conduction of heat through the first thermaltransfer plate and the second thermal transfer plate, respectively, tothe liquid-cooled structure via the first curved extension and thesecond curved extension.
 13. The cooled electronic system of claim 4,wherein the first thermally conductive extension and the secondthermally conductive extension each physically attach to the uppersurface of the liquid-cooled structure via, in part, contactingrespective opposing fins of a plurality of opposing fins extending fromthe upper surface of the liquid-cooled structure.
 14. The cooledelectronic system of claim 4, wherein the thermal spreader furthercomprises a connecting flange connecting the first thermally conductiveextension and the second thermally conductive extension where physicallyattached to the upper surface of the liquid-cooled structure, andwherein the liquid-cooled structure comprises an elongate slot structuredisposed at the upper surface thereof, the connecting flange extendinginto the elongate slot structure disposed at the upper surface of theliquid-cooled structure.
 15. A liquid-cooled electronics rackcomprising: an electronics rack comprising an electronic assembly, theelectronic assembly comprising: an electronics card comprising a firstsurface and a second surface on opposite sides of the electronics card;a socket with a latch at one end of the socket, the latch facilitatinglatching of the electronics card within the socket or unlatching of theelectronics card from the socket; a cooling apparatus, the coolingapparatus comprising: a liquid-cooled structure disposed at the one endof the socket, the liquid-cooled structure comprising a thermallyconductive structure with at least one coolant-carrying channelextending therein; and a thermal spreader coupling the electronics cardto the liquid-cooled structure and facilitating the conduction of heatfrom the electronics card to the liquid-cooled structure, the thermalspreader comprising: a first thermal transfer plate and a second thermaltransfer plate, the first thermal transfer plate comprising a firstthermal conduction surface and the second thermal transfer platecomprising a second thermal conduction surface, the first thermalconduction surface and the second thermal conduction surface being inspaced, opposing relation and configured to accommodate the electronicscard therebetween with the first thermal conduction surface coupled tothe first surface of the electronics card and the second thermalconduction surface coupled to the second surface of the electronicscard, and wherein the first thermal transfer plate and the secondthermal transfer plate each comprises a first end edge and a second endedge; and a first thermally conductive extension extending from thefirst end edge of the first thermal transfer plate, and a secondthermally conductive extension extending from one of the first end edgeor the second end edge of the second thermal transfer plate, the firstthermally conductive extension coupling the first thermal transfer plateto the liquid-cooled structure, and wherein the first thermallyconductive extension is disposed at a first side of the latch at the oneend of the socket of the electronic assembly, the latch comprisingopposite first and second sides, and wherein the latch is accessible andfacilitates securing the electronics card within the socket or removalof the electronics card from the socket without removing at least one ofthe liquid-cooled structure or the thermal spreader.
 16. Theliquid-cooled electronics rack of claim 15, further comprising anotherliquid-cooled structure disposed at another end of the socket, andwherein the second thermally conductive extension extends from thesecond end edge of the second thermal transfer plate and couples thesecond thermal transfer plate to the another liquid-cooled structure.17. The liquid-cooled electronics rack of claim 15, wherein the secondthermally conductive extension extends from the first end edge of thesecond thermal transfer plate and couples the second thermal transferplate to the liquid-cooled structure, and wherein the second thermallyconductive extension is disposed at the second side of the latch at theone end of the socket of the electronic assembly.
 18. The liquid-cooledelectronics rack of claim 17, further comprising another liquid-cooledstructure disposed at another end of the socket, and wherein the thermalspreader further comprises a third thermally conductive extensionextending from the second end edge of the first thermal transfer plateand a fourth thermally conductive extension extending from the secondend edge of the second thermal transfer plate, the third thermallyconductive extension coupling the first thermal transfer plate to theanother liquid-cooled structure, and the fourth thermally conductiveextension coupling the second thermal transfer plate to the anotherliquid-cooled structure, and wherein the latch at the one end of thesocket is a first latch, and the socket further comprises a second latchat the another end of the socket, the second latch comprising oppositefirst and second sides, and wherein the third thermally conductiveextension is disposed at the first side of the second latch, and thefourth thermally conductive extension is disposed at the second side ofthe second latch, wherein the electronics card is securable within thesocket or removable from the socket using the first latch and the secondlatch without requiring removal of the liquid-cooled structure, theanother liquid-cooled structure, or the thermal spreader.
 19. Theliquid-cooled electronics rack of claim 17, wherein the first thermallyconductive extension extending from the first thermal transfer plate andthe second thermally conductive extension extending from the secondthermal transfer plate physically couple to an upper surface of theliquid-cooled structure, and wherein the first thermally conductiveextension extending from the first thermal transfer plate and the secondthermally conductive extension extending from the second thermaltransfer plate each comprise a flange extending from an end thereofphysically coupled to the upper surface of the liquid-cooled structure,each flange facilitating physical coupling of the respective firstthermal transfer plate and second thermal transfer plate to the uppersurface of the liquid-cooled structure.
 20. (canceled)